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ABSTRACT 

This study explored the mechanics of using an enhanced, 
comprehensive multipurpose logic model, the Performance Blueprint, as a means 
of building evaluation capacity, referred to in this paper as performance 
measurement literacy, to facilitate the attainment of both service-delivery 
oriented and community-oriented outcomes. The application of this approach is 
illustrated in a short case study. By performance measurement literacy is 
meant the capacity of individuals and institutions to obtain, interpret, and 
understand performance measurement information and the competence to use such 
information to benefit clients, service delivery, and the community (P. Long, 
2002) . In the Performance Blueprint, the expanded logic model used by the 
Appalachian Partnership for Welfare Reform (APWR) , the approach revolves 
around a representational depiction of an initiative's intended logic, or how 
inputs are expected to produce outcomes. The Blueprint makes distinctions 
between effort and effect and between output and outcome. The case study 
shows the application of the Performance Blueprint to enhance performance 
measurement literacy in the provision of training and technical assistance to 
APWR partners working to meet the needs of county-level agencies. One 
appendix defines andoperationalizes performance measurement literacy, and the 
other summarizes questions and considerations related to the four quadrants 
defined by M. A. Friedman (1997). (Contains 22 references.) (SLD) 
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INTRODUCTION 

In this paper I will explore the mechanics of using an enhanced, comprehensive, 
multipurpose logic model, called the Performance Blueprint, as a means of building 
evaluation capacity, referred to in this paper as performance measurement literacy, to 
facilitate the attainment of both service-delivery-oriented and community-oriented 
outcomes. To do this I will first need to define some terms and operationalize some 
constructs, whose application I then hope to illustrate in a short case study. 

CONTEXT 

Before examining the terms and constructs to be used in this paper, I should say a 
word or two about the setting in which they have their most practical significance. I have 
been the “outcome evaluator” of the Appalachian Partnership for Welfare Reform 
(APWR) since its inception in 1998. The APWR is a continuing collaboration funded by 
the Ohio Department of Job and Family Services (ODJFS), endorsed by the directors in 
the Ohio Appalachia region and administered by the Institute for Local Government 
Administration and Rural Development (ILGARD) at Ohio University’s Voinovich 
Center for Leadership and Public Affairs. The partnership is intended to facilitate the 
achievement of federal, state and local welfare reform and workforce development goals 
throughout the region by building and enhancing administrative infrastructure and 
capacity among the three APWR partner groups: 1 .) the twenty-nine Appalachian County 
Job and Family Service Agencies, 2) ODJFS central office and regional representatives, 
and 3) ILGARD. With the assistance of talented colleagues,' I am frequently engaged in 
challenges directly and indirectly related to what Baizerman and co-editors call 
“evaluation capacity building,” which they define as “the intentional work to 
continuously create and sustain overall organizational processes that make quality 
evaluation and its uses routine (Baizerman, et al . 2002:1).”^ The emphasis on this type 
of capacity building is central to the APWR’s Ongoing Performance Measurement and 
Management (OPM&M) model, components of which are provided to the partners 
through many of our APWR products and services. These include: Community-Based 
Welfare-Reform Strategic Planning, Performance-Based Contract Management, 
Performance-Based Outcomes Management, Workforce Investment Act (WIA) 
Construction Kit, Ongoing Plan Management (0PM), and the Social Marketing Initiative, 
all of which can be viewed at http://www.ilgard.ohiou.edu/apwr/ . 

TERMS AND CONSTRUCTS 

Performance Measurement v. Program Evaluation 

Within the public-sector setting of the APWR, building evaluation capacity has 
required us to differentiate between “performance measurement” and “program 
evaluation.” The U.S. General Accounting Office (US GAO 1998; see also Hatry 1999) 
provides a useful and authoritative definition of “performance measurement” and 
contrasts it with “program evaluation.” Table 1 outlines some of the important 
similarities and differences. 



’ Ms. Stephanie Howe, APWR Program Manager, as well as Cindy Poole, Dr. Chris Miewald, Karl Runser, 
and Dr. Barry Oches make up the APWR service delivery team. 

^ M. Baizerman, D. W. Compton, and S.H. Stockdill are the editors of the Spring 2002, Number 93 issue of 
New Directions for Evaluation entitled, “The Art, Craft, and Science of Evaluation Capacity Building.” 
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Table 1 

Similarities and Differences between 
Program Evaluation and Performance Measurement 





Program Evaluation (PE) 


Performance Measurement (PM) 


Definition: 


periodic or ad hoc studies to assess 
how well a program is working 


ongoing monitoring and reporting of 
program accomplishments, 
particularly progress towards 
preestablished goals 


When is it 
conducted? 


on a periodic or ad hoc basis 


on an ongoing basis 


Who conducts 
it? 


experts external to the program, 
either inside or outside the agency, 
as well as by program managers 


program or agency management 


Focus: 


PE focuses on broad range of 
information on program performance 
and its context 


PM focuses on whether a program 
has achieved its objectives, 
expressed as measurable 
performance standards 


Scope: 


in-depth examination of program 
performance and context allowing 
for an overall assessment of whether 
the program works and identification 
of adjustments that may improve its 
results 


PM, because of its ongoing nature, 
can serve as an early warning system 
to management and as a vehicle for 
improving accountability to the 
public. 



Source: U.S. General Accounting Office (US GAO 1998) 



These clarifications have helped the APWR to develop and provide applicable products 
and services that focus on building much needed “performance measurement” capacity. 
Accordingly, we make use of a practical definition of performance measurement that is 
consistent with the GAO’s definition and reflective of current efforts to shed light on 
evaluation capacity building. Performance measurement involves 1) systematically 
collecting and strategically using information, 2) on an ongoing basis, 3) in an intra- and 
inter-organizational fashion, and 4) for a variety of internal and external purposes. An 
explanatory outline for this definition can be found at the website provided above.^ 

Performance Measurement Literacy 

In an effort to better understand the capacities and competencies associated with 
performance measurement, we have constructed the concept of performance- 
measurement literacy. By performance measurement literacy we mean “the capacity of 
individuals and institutions to obtain, interpret, and understand performance measurement 
information and the competence to use such information to benefit clients, service 
delivery, and the entire community (Longo 2002).” Performance measurement literacy 
encompasses a wide range of technical and cultural competencies associated with the 



^ This comprehensive definition of performance measurement is more closely related to the formal 
definition of evaluation capacity building, as found in Stockdill and others (2002:8), which is, “a context- 
dependent, intentional action system of guided processes and practices for bringing about and sustaining a 
state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing 
practices within and/or between one or more organizations/programs/sites.” 
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systematic collection and strategic utilization of information; however, inclusive, 
representative, and collaborative stakeholder involvement is a key ingredient for these 
practices to become fully institutionalized and to result in effective leadership, stronger 
and more productive communities, and engaged citizens (Burke 1998, Clegg & Associates 
1999, Fetterman 2001, Gaventa, et al . 1998, and Preskill et al . 2000). Appendix A 
contains a more detailed explanation of the performance measurement literacy construct 
along with examples of the four inter-related skill sets. We will discuss this construct in 
the context of the case study at the end of the paper. 

The Performance Blueprint, APWR’s Expanded Logic Model 

Like many other approaches rooted in “theory-based” or “theory of change- 
based” evaluation traditions (Chen & Rossi 1983, Connell & Kubisch 1996; see also 
Patton 1997: 215-238), the APWR’s OPM&M model revolves around a representational 
depiction of an initiative’s intended logic, i.e., how inputs are expected to produce 
outcomes. OPM&M makes use of an expanded and innovative logic model, called the 
“Performance Blueprint,” as both a planning and evaluation tool."* The “Blueprint” is 
enhanced and strengthened by 1.) an anthropological orientation (Longo 2001b, Longo & 
Miewald 2000); 2.) a social marketing orientation (Brown 1997, Bryant, et al . 2001, 
Lefebvre & Floral988, and Sutton, et al . 1995); and 3.) Mark Friedman’s Four-Quadrant 
Approach to Performance Measurement (Friedman 1997, Friedman, et al. 2000). These 
enhancements have helped the Blueprint exceed the limitations often cited with regard to 
the use of logic models, most of which focus on how the logic model amounts to little 
more than a “Procrustean bed” (Stufflebeam 2001:39) that pays little or no attention to 
the underlying sociocultural and political variables associated with the program’s 
“context” or “environment” (Fisher 2001, Freddolino, et al. 1998, Perrin 1998, and 
EVALTALK 2002). In addition to the traditional elements encompassed by most 
standard logic models (e.g., inputs, activities, outputs, and outcomes), the Performance 
Blueprint also requires the identification of direct and indirect beneficiaries (clients, 
customers, information users, etc.) as well as the direct and indirect service providers 
(vendors, subcontractors, collaborators, etc.). Moreover, by incorporating Friedman’s 
Four-Quadrant Approach, the Blueprint also offers a transparent strategy for identifying 
and prioritizing four types of performance measures associated with a program’s effort- 
related outputs and its effect-related outputs, each of which is further divided into 
quantity (individual counts) and quality (rate and percentages). This “effort-effect 
distinction” and its impact on the Performance Blueprint will be discussed in the next 
section. Closely related to the effort-effect distinction is a parallel distinction that allows 
the Blueprint to distinguish between Service-Delivery Outcomes (effort-related) and 
Community Outcomes (effect related). This will be discussed in more detail below. 



Logic models are visual depictions of what a program can be expected to produce. They have also been 
called results chains, outcome-sequence charts, strategy maps, and so forth. Wholey (1979) is one of the 
first to make use of the device and the term. See also Julian, et al. (1995), Julian (1997), McLaughlin & 
Jordan 1999, Plantz, et al. (1997), United Way of America (1996), and Wholey (1979 & 1997). 
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THE EFFORT-EFFECT DISTINCTION AND HOW IT ENHANCES THE 
PERFORMANCE BLUEPRINT 

The Performance Blueprint makes use of Friedman’s four-quadrant approach as a 
device for understanding and organizing OUTPUTS. This is illustrated in Figure 1. 
Before discussing how this insertion sheds light on the scope and scale of OUTCOMES, 
we will first examine the mechanics and usefulness of distinguishing between “measures 
of effort” and “measures of effect.” 

Figure 1 

The Performance Blueprint 




I should point out that Friedman does not make use of a standard, four-part logic model 
(e.g., inputs, activities, outputs, and outcomes). Rather, in his Four Quadrant Approach 
to organizing program performance measures, as depicted in Figure 2, he associates 
measures of effort with “input” (i.e., process or service delivered) and measures of effect 
with “output” (i.e., product or client condition achieved). For all practical purposes, 
Friedman uses the Four-Quadrant Approach as a self-contained logic model; whereas, the 
Performance Blueprint embeds this four-quadrant schematic into the traditional, linear 
logic model chain. It should be evident, therefore, why the insertion of this dynamic 
schematic would have a ripple effect on the other elements of the traditional, linear logic 
model before and after its insertion. 



Figure 2 

Friedman’s Four Quadrant Approach to Program Performance Measures 
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In Friedman’s view, which assumes that the reader is a program staff person, measures of 
effort answer the question, how hard did we try! Measures of effect, then, answer the 
question, what changes did we produce'} Notice that for each of these two types of 
measures there is an additional pair of questions that helps to distinguish between 
quantity (how much did we do?) and quality (how well did we do it?). Note also that 
“quantity” (#) will usually represent a number; while “quality” (%), a percentage or a rate 
of some sort. In this way Friedman offers logical and transparent criteria that make it 
possible to prioritize and rank the four distinct types of measures: 1.) measures of the 
quality of the effect; 2.) measures of the quality of the effort; 3.) measures of the quantity 
of the effect; and 4.) measures of the quantity of effort. A very revealing set of 
mathematical relationships emerges when the appropriate information is placed into the 
appropriate quadrant. This paper is not the proper context in which to elaborate on these 
mathematical phenomena; however, it must be pointed out that, without such 
foundational information (i.e., the correct denominator) as that which is contained in the 
fourth-priority quadrant (e.g., number of people served), it would be impossible to 
achieve any accuracy in calculating any first-priority quadrant information (e.g., the 
percentage of targeted/reached clients who actually benefited). 

I mentioned that the effort-effect distinction has had a ripple effect on the logic 
and, therefore, potential usefulness of the entire Performance Blueprint. Figure 3 
attempts to capture this impact. The dotted line running through the Blueprint 



Figure 3 

How the Effort-Effect Distinction Echoes throughout the Performance Blueprint 



Inputs Activities/Strategies Outputs Outcomes 



I . Service Providers, 

I Collaborators, Vendors, etc. 





Material & nonmaterial 
inputs to operate 
internally-oriented 
activities. 


4 


2 

Internally-Oriented Activities 
(e.g., contract management, quality 
assurance, monitoring & reporting, 
performance measurement, etc. ) 


4 
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4 
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Material & nonmaterial 
inputs to operate 
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4 
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4 


Community 

Outcomes 

1 





I Clients, Customers, (and 
I information users) 



corresponds to the effort-effect distinction and demonstrates that, if one were to 
distinguish between performance measures of effort and performance measures of effect 
in Friedman’s manner , one could apply the same distinction to Inputs, Activities, and 
Outcomes. In training sessions we have found it helpful to point out that performance 
measures of effort and service-delivery outcomes reflect the “performance” of service 
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providers, collaborators, and vendors. Likewise, the performance measures of effect and 
community outcomes reflect the “performance” of reached clients and customers.^ 

Appendix B contains some more detailed questions and considerations for 
identifying and prioritizing performance measures in a manner consistent with 
Friedman’s approach. Table 2 shows a few quick examples. 



Table 2 

Measuring Outputs (Performance Measures) 



MEASURES of EFFORT 


Quantity (#) 


Quality (%) 


4^* Priority 


2^ Priority 


The number of services delivered. 


The percentage of service delivery provided 


The (numerical) amount of effort put into 


well (efficiently &. equitably), (e.g., % of hard- 


service delivery. 


to-reach customers reached) 

The percentage of satisfied customers. 


MEASURES of EFFECT 


Quantity (#) 


Quality (%) 


Priority 


Priority 


The number of clients or customers who 


The percentage of clients or customers who 


showed an improvement in well being. 


showed an improvement in well being. 



THE OUTPUT-OUTCOME DISTINCTION WITHIN THE PERFORMANCE 
BLUEPRINT 

Friedman’s approach, particularly the four types of questions he raises in his 
schematic, calls attention to the immediate and direct performance of resources, planning 
strategists and strategies, and service-delivery personnel in relation to clients and/or 
customers who are targeted and reached. This focus has helped our team define and 
operationalize “outputs” in a way that is much narrower in scope and scale than those of 
“outcomes.” Outputs, accordingly, refer to specific impacts that service providers 
actually have on the clients/customers that are reached. The “performance” of service 
providers is gauged by examining their outputs, in terms of how efficiently and how 
effectively they were able to reach and satisfy their clients’/customers’ needs and the 
extent to which successes along these lines can be associated with the attainment of the 
overall desired outcomes of the community. 



* In a previous rendition of the Performance Blueprint I had placed “clienu and customers” in the box 
above Activities/Strategies and “service providers, collaborators, and vendors” under Activities/Strategies. 
I am grateful to my colleague. Dr. Barry Oches, for pointing out how much more instructionally coherent it 
would be to align these players with the effort-effect distinction. 
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Outcomes, in this view, can be understood as the sought-after improvements in 
the general conditions of the broader community in which “reached” clients and 
customers live. This may mean the town in which they live, the county, the state, and so 
forth. At any rate, it is the difference in scale between the whole population and the 
population actually served. Outcomes are the consequences of outputs, the broad impact 
of all service delivery on the whole community’s development. Community- wide 
indicators are the proxy measures of the attainment over time of community outcomes; 
these can include statistics related to economic prosperity, health, education, safety, 
recreation and so forth. 

As outputs can be categorized into measures of effort and measures of effect, so 
too can outcomes be categorized. Figure 4 attempts to illustrate the extension of these 
categories. 

Figure 4 

Outcomes as Extensions of Effort-Related and Effect-Related Outputs 




In the obvious heterodoxy that characterizes the use of terms and concepts in 
outcomes-based performance measurement/management it has been helpful for our team 
to frame outcomes in this way. Our training participants have commented that it makes 
sense to distinguish between effort and effect in relation to both outputs and outcomes. 
Figure 5 shows an application of this logic to the way Outcomes have been defined by 
Ohio Department of Job and Family Services (ODJFS).^ The attainment 



* See ODJFS web page: http://www.state.oh. us/odifs/aboutus/mission.stm . 
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Figure 5 

An Application of Outcomes Categorized as Extensions of 
Effort-Related and Effect-Related Outputs: Ohio Department of Job and Family Services 




of Outcome 5, even though it is last in a list of five outcomes, would surely be a 
reflection and a consequence of the effort of the entire system’s personnel, its leadership, 
and its organizational culture. Without a serious organizational commitment to these 
internally oriented, service-delivery outcomes, about which all stakeholders — especially 
taxpayers — have a right to be concerned, few if any externally oriented, community- 
based outcomes could ever be achieved. 

USING THE PERFORMANCE BLUEPRINT TO ENHANCE PERFORMANCE 
MEASUREMENT LITERACY: DISCUSSION AND SHORT CASE STUDY 

The APWR service-delivery team has been providing training and technical- 
assistance services to APWR partners within and beyond Appalachian Ohio for nearly 
four and a half years. Many of these services make use of the Performance Blueprint in 
our general APWR effort to build evaluation capacity, which as I mentioned above, we 
describe as enhancing “performance measurement literacy.” Based on general participant 
feedback from the different types of performance-measurement-related trainings that the 
team provides, we have discovered that our way of organizing the Performance Blueprint 
has helped county agency personnel — as well as some ODJFS personnel — better 
understand and exceed many of the often-cited limitations surrounding the establishment 
of performance measurement practices (See Fisher 2001 and Perrin 1998 for example). 
There is some evidence to suggest a connection between the Performance Blueprint and 
gains in performance measurement literacy among the participants we have trained.^ I 
would like to illustrate this by using one of the APWR product/service lines that revolves 
around the Performance Blueprint, APWR Performance-Based Contract Management.* * 



’ Each year the APWR conducts an annual performance assessment. This year’s report, the Fourth Annual 
Performance Assessment, and previous years’ report are available on the APWR website at 
http://www.ilgard.ohiou.edu/apwr/ . Participant feedback is furnished in these reports. 

* To assist me in this illustration I will draw on a follow-up survey completed by trainees six months after 
training and a summary that Karl Runser prepared for the Fourth Annual APWR Performance Assessment 
released in October 2002. 
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APWR Performance-Based Contract Management 

The APWR’s Performance-Based Contract Management products and services 
are offered to meet the contract management needs of county-level agencies. We 
facilitate specialized executive leadership seminars, general training workshops, and a 
three-part series of specialized follow-up roundtable discussions. At the center of the 
training curriculum are numerous hands-on activities that give participants many 
opportunities to become familiar with the theoretical and practical features of the 
Blueprint along with accompanying templates and forms. Participants can use hardcopies 
of the Contract Management Manual, which furnishes photocopy-ready templates and 
forms or they can use the APWR website, which allows users to download the templates 
and forms as Word and WordPerfect documents and Excel spreadsheets. 

According to follow-up feedback, training in Performance-Based Contract 
Management has helped participants move closer towards the attainment of the following 
general service-delivery outcomes: regulatory compliance, welfare reform and changing 
the community perception of the “welfare” agency, cost saving, operating efficiency, 
increased accountability, consistent application of policies, and movement toward a 
performance centered model. Accordingly, progress in these areas has made it more 
possible for agencies to demonstrate their contributions to workforce and economic 
development and poverty alleviation in general. 

Participants also report that training on and exposure to the Performance 
Blueprint have enabled them to view the establishment of performance-based contract 
management practices from a “systems” perspective encompassing more than simply the 
legalistic properties of managing contracts. Of all the products and services offered by 
the APWR, performance-based contract management generated the highest score when it 
came to impacts related to “enhancing intra- and inter-organizatiojial collaboration.” 
Many participants reported that, by having the evaluation criteria presented explicitly and 
logically by means of the “Quads” from within the Performance Blueprint in relation to 
the other components, such as service providers, intended clients, and overarching 
outcomes, managing contracts has become less mystifying. Others report that, because of 
the contextualized focus on “systems thinking” in relation to the community’s desired 
outcomes, RFPs are now being written with performance reporting and relationship 
building in mind. 

One individual, who performs contract management and monitoring services 
under contract with two Ohio Appalachian County Departments of Job and Family 
Services, made the following statement, which highlights many of his colleagues’ 
feedback: 



In both counties, we have adopted the methods and terminology of 
the APWR manual. We constantly refer to the manual as a guide for 
our contract management. The manual has been extremely effective. 
It helps to have a comprehensive plan for contract management as 
described in the manual. The Directors then know exactly where 
we’re going with the process. And contract providers know ‘up front’ 
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what the expectations of the [agency] are. There are few ‘surprises’ 
in regard to performance expectations and measurements that emerge 
after a contract is signed. 

Another respondent, an Assistant County Administrator, reported that, “the information 
and techniques learned during the training helped us to greatly improve our contracting 
process.” He also mentioned that in contrast to the “haphazard” contracting and 
monitoring system of the past, his agency now has “a contracting process, monitoring 
system and contract template that give us better control over the services we are 
purchasing.... Thanks to the training we can now better manage the contractors since all 
requirements are clearly spelled out in the document.” 

Several respondents brought up the need for updating the manual and the forms to 
account for changes in policies and regulations. They called for follow-up training 
sessions as well, and not only for agency personnel, but also for vendors so that there 
would be even greater agreement and understanding among contracting parties 
throughout all phases of the process, i.e., from RFP construction through contract close- 
out. 



These observations directly and indirectly touch on the four skill sets brought to 
light by the performance measurement literacy construct. Having previously provided 
general training and technical assistance in performance measurement to local agencies in 
relation to strategic planning, board development, and other multipurpose forms of 
outcomes and performance management, the team was somewhat surprised to discover 
that our contract management products and services were becoming well received and 
popular. After all, contract management has never been known as a source of exciting 
opportunities to demystify an organization’s mission, to build collaborative partnerships, 
to become a learning organization, and to document the achievement of results and 
accomplishments that usually go unnoticed. Granted, not all of this positive reception 
and popularity can be attributed to the Performance Blueprint or the performance 
measurement literacy construct. However, these devices did help agency personnel 
recognize that, in fact, they more frequently exercise skills from Set #1 and less 
frequently skills from Sets #2, #3, and #4 as shown in Appendix A. 

For far too long agency personnel have been required to do little more than collect 
information that then disappears before having local applicability. The performance 
measurement literacy construct helps to identify two important needs: 1 .) the need for 
political/organizational/cultural skills in relation to both collecting and using performance 
information, and 2.) the need for both technical/statistical and organizational/cultural 
skills in the general area of strategically using performance information. Successes along 
either of these two lines will converge with current efforts aimed at building and 
cultivating learning and/or evaluation cultures (Davidson 2001, Wenger 1998, and 
Trochim 1999, Longo 2001b, and Lon go & Miewald 2000). 
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Conclusion 

The value of the Performance Blueprint has begun to become evident by virtue of 
the fact that a small but growing number of agencies have begun to institute practices 
amounting to changes in their standard operating procedures and, in turn, their 
organizational cultures. Some of this is due to the socio-technical mechanics within the 
Performance Blueprint that allow it to operate as a planning and an evaluation tool. 
However, in and of themselves, tools are lifeless, inactive objects and, therefore, 
incapable of making a difference. People in organizational settings must use 
appropriately customized yet standardized tools to make any kind of sustainable 
difference, whether that means purposefully maintaining a value or adding value by 
bringing about change or managing transitions already set in motion by other changes. 
For this reason, as we present the Performance Blueprint as a tool, we also present the 
holistic performance measurement literacy construct as an assortment of skills aligned 
with the tasks associated with instituting performance measurement practices. These 
tools were developed to facilitate a better understanding of the differences between 
measures of effort and measures of effect, the scope of outputs and the scope of 
outcomes, the technical and cultural skills needed to collect and use performance 
information. Performance measurement literacy combines know-how and do-how for 
the benefit of those being served as well as those doing the serving. 
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Appendix A 

Performance Measurement Literacy DeHned and Operationalized 



Linked to our working definition of performance measurement (systematically collecting & strategically 
utilizing information on an ongoing basis in an intra- and inter-organizational fashion for a variety of 
internal and external purposes), we have defined Performance Measurement Literacy is a.) the capacity of 
individuals and institutions to obtain, interpret, and understand performance measurement information and 
b.) the competence to use such information to benefit clients, service delivery, and the entire community. 
Figure 6 and its accompanying chart illustrate our working definition of performance measurement literacy, 
which is divided into four sets of skills that are determined by the intersection of a collecting - using 
information skills continuum and a technical - cultural skills continuum. 

Figure 6 

The Four Skill Sets of Performance Measurement Literacy 



COLLECTING INFORMATION 
Skills Associated with Systematically Collecting Information 



“TECHNICAL” 
Statistical & 
Technical Skills 




“CULTURAL” 
Political, Social &. 
Organizational Skills 



USING INFORMATION 

Skills Associated with Strategically Utilizing Information 



Set 1 contains technical/statistical skills in the 
SYSTEMATIC COLLECTION of performance 
information. 


Set 2 contains the cultural (political, social, and 
organizational) skills in the SYSTEMATIC 
COLLECTION of performance information 


• Identifying available, reliable, and useful information 
sources and indicators that will help to monitor progress 

• Using databases and technology appropriately to 
collect and maintain meaningful data and information 

• Utilizing available information sources and accessible 
indicators to specify baseline data (i.e., statistical 
information that can be used as a starting point) 

• Selecting and targeting a manageable range of 
indicators that will be tracked and used to gauge progress 


• Defining and prioritizing stakeholders’ definitions 
of success (i.e., required and desired outcomes) 

• Anticipating and accounting for the information 
needs of multiple internal and external stakeholders 

• Gathering information from multiple internal and 
external sources on a routine basis 

• Focusing on expected changes in target 
population(s) and/or target social condition(s) (i.e., 
outcomes as new knowledge, increased skills, changed 
attitudes & values, modified behaviors, altered statuses, 
improved conditions, etc.) 


Set 3 contains technical/statistical skills in the 
STRATEGIC UTILIZATION of performance 
information. 


Set 4 contains the cultural (political, social, and 
organizational) skills in the STRATEGIC 
UTILIZATION of performance information. 


• Using the appropriate methodology to analyze 
qualitative and quantitative information 

• Displaying performance measurement information in 
ways that are clear and meaningful to the audience (this 
may vary depending upon the audience — their interest, 
background and literacy skills) 

• Generating reports to communicate whether and/or to 
what extent the data and information being tracked meets 
targeted expectations 


• Developing ongoing formative and summative 
strategies to make use of qualitative and quantitative 
research and evaluation techniques in response to the 
needs of newly emerging target populations 

• Routinely sharing information and data with local 
stakeholders 

• Summarizing the periodic findings into 
recommendations for decision makers and stakeholders 

• Discovering new applications of performance 
measurement information 
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Appendix B 

Questions and Considerations Related to Friedman’s Four Quadrants 



1 MEASURES ofEEFORT 


Quantity (#) 


Quality (%) 


4“ Priority 


2"^ Priority 


HOW MUCH DO STAFF MEMBERS PRODUCE 


HOW WELL DO STAFF PROVIDE SERVICES 


OR DELIVER? 


AND PERFORM ACTTVITIES? 


HOW MUCH DOES A PARTICULAR PROJECT 


HOW WELL ARE PROJECTS IMPLEMENTED? 


PRODUCE OR DELIVER? 


HOW WELL DO STAFF PERFORM THESE 


# Clients/customers served 


AcnvrriEs? 


Are there better, more specific ways to 


HOW WELL ARE STAFF PREPARED FOR THE 


count customers or important subcategories 


JOB(S)? 


of customers, and list them? (e.g. # of 


% Common Measures: 


families served, # of children with 


Client-staff ratio 


disabilities served, # of businesses served. 


Staff turnover rate 


etc.). 


Staff morale 
Fully trained staff 


# Activities performed/Services provided 


% Satisfied customers 


What kind (how many) of activities are 


% Hard-to-reach customers served 


performed? 


% Activity-specific Measures: 


What kind (how many) of services are 


% Services delivered on time 


delivered? 


% Clients completing activity 


Convert each activity/service into a measure 


% Activities performed correctly and 


(e.g. "we train people" becomes # of people 


completely 


trained; “we serve businesses” becomes # of 


% Actions meeting other objective, service- 


businesses served, etc.) 


delivery standards or criteria 


ME A S URES of EFFECT 


Quantity (#) 


Quality (%) 


Priority 


J“ Priority 


IS ANYONE BETTER OFF (Individual case 


IS ANYONE BETTER OFF (percentages and 


counts)? 


rates)? 


Guiding questions: 


Guiding questions: 


In what ways could clients be better off as a result 


In what ways could clients be better off as a result 


of utilizing this service? 


of getting this service? 


How would anyone know if clients/customers were 


How would anyone know if clients/customers were 


better off in measurable terms? 


better off in measurable terms? 


# of clients/customers with increases in: 


% of clients/customers with increases in: 


• Skills/Knowledge (e.g., parenting skills) 


• Skills/Knowledge (e.g., parenting skills) 


• Attitude (e.g., towards drugs) 


• Attitude (e.g., towards drugs) 


• Behavior (e.g., school attendance) 


• Behavior (e.g., school attendance) 


• Status/Circumstances (e.g., # entered 


• Status/Circumstances (e.g., entered 


employment, # earnings increase, # in stable 


employment rate, % earnings increase, % in 


housing/living arrangement) 


stable housing/living arrangement) 



Adaptation of (Friedman 2000) 
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